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Abstract 


This  paper  considers  the  application  of  Bayesian  techniques  for  simul¬ 
taneous  estimation  to  the  specification  of  regression  weights  for  selection 
tests  used  in  various  technical  training  courses  in  the  Narine  Corps. 
Results  of  a  method  for  ar-group  regression  developed  by  Molenaar  and  Levis 
(1979)  suggest  that  common  weights  for  training  courses  belonging  to  certain 
general  categories  are  justified  in  many  cases.  However,  such  commonality 
of  regression  weights  does  not  appear  to  hold  for  all  courses  in  these 
categories — weights  for  some  training  courses  remain  distinct  even  after  the 
application  of  the  simultaneous  estimation  procedure.  Thus,  a  hypothesis  of 
validity  generalization  across  training  courses  in  a  given  category  would 
ouly  be  retained  for  a  carefully  selected  subset  of  courses  and  not  for  all 
groups  included  in  the  analyses - ■  .t  fO-c  ■],.  ,, 
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Sinnlttneoas  Estimation  of  Regression  Functions 
for  Marine  Corps  Technical  Training  Specialties 

The  relative  valne  of  a  regression  function  for  predicting  future 
performance  is  related  to  its  consistency  of  prediction  in  important  sub¬ 
groups  of  examinees.  When  large  differences  between  predictor-criterion 
relationships  exist  for  distinct  subpopulations  of  interest,  the  use  of  a 
common  prediction  equation  is  questionable  for  a  variety  of  reasons.  This 
perspective  reiterates  a  historical  concern  for  comparisons  of  more  than 
overall  predictor-criterion  correlations  in  validation  research.  As  noted 
by  Humphreys  (1952),  useful  subgroup  comparisons  must  ask  whether  the  same 
score  has  the  same  meaning  in  the  groups  being  compared,  i.e.  whether  the 
regression  lines  are  identical  or  merely  parallel  (p.  134).  One  would  only 
add  to  this  an  obvious  concern  for  subgroup  regressions  that  are  neither 
identical  nor  parallel. 

Empirical  comparisons  of  regression  equations  for  subgroups  defined  by 
demographic  variables  such  as  gender  and  race  have  generally  followed  proce¬ 
dures  first  outlined  by  Gulliksen  and  Wilks  (1950)  for  statistical  tests  of 
the  equality  of  errors  of  estimate,  slopes  and  intercepts.  When  interest 
focuses  on  regions  of  the  predictor  space  where  the  degree  of  differences 
between  regressions  is  acute,  the  Johnson-Neyman  technique  has  also  been 
employed  (see  Gamache  and  Novick,  1985  and  Dunbar  and  Novick,  1985  for  some 
recent  examples).  Regression  comparisons  performed  by  these  techniques  are 
perhaps  well  suited  for  settings  involving  a  small  number  of  groups,  al¬ 
though  they  are  by  no  means  limited  to  such  settings. 

An  alternative  approach  to  accommodating  differences  among  subgroups  in 
regression  analysis  is  found  in  the  literature  on  central  prediction 
systems.  Procedures  such  as  those  reviewed  by  Linn  (1966)  address  the 


problem  of  making  adjustments  to  predictor  and  criterion  scores  for  in¬ 
dividuals  of  varying  subgroups  sucb  that  overall  accuracy  of  prediction  is 
increased  upon  cross-validation.  A  limiting  case  for  approaches  such  as 
these  is  Cleary's  (1966)  individual  differences  model  for  multiple 
regression.  As  discussed  by  Linn  (1966)  and  others,  however,  empirical 
studies  of  systems  for  central  prediction  have  indicated  little  promise — 
perhaps  because  each  classical  procedure  posits  a  particular  model  of  group 
differences  by  the  nature  of  the  adjustments  that  are  made  to  predictors  and 
criteria.  Model  restrictions  imposed  by  one  central  prediction  system  may 
not  be  justified  for  all  groups  belonging  to  the  system  (Novick  and  Jachson, 
1974)  and  the  effectiveness  of  the  complete  system  is  likely  to  be  com¬ 
promised  as  a  result.  In  such  cases  a  more  flexible  model  for  prediction  in 
the  multiple-group  situation  is  advised. 

The  purpose  of  this  paper  is  to  describe  the  method  of  Bayesian  simul¬ 
taneous  estimation  of  multiple  regression  in  m-groups  and  to  illustrate  the 
application  of  this  method  to  the  problem  of  specifying  prediction  weights 
for  subtests  of  the  Armed  Services  Vocational  Aptitude  Battery  (ASVAB), 
Forms  6  and  7,  in  a  variety  of  technical  training  specialties  in  the 
military.  The  general  approach  to  this  problem  was  first  developed  by 
Lindley  (1971)  and  Lindley  and  Smith  (1972),  and  further  refined  and  applied 
by  Novick,  Jackson,  Thayer,  and  Cole  (1972),  who  demonstrated  empirically 
the  effectiveness  of  this  method  in  increasing  predictability.  The  par¬ 
ticular  model  adopted  in  this  paper  is  due  to  Nolenaar  and  Lewis  (1979),  who 
developed  it  as  a  refinement  of  earlier  procedures  noted  above.  Other 
approaches  to  the  problem  have  been  implemented  by  Rubin  (1980)  and  Braun, 
Jones,  Rubin,  and  Thayer  (1982). 


Model  Specification 


The  model  for  mnltiple  regression  in  m-gronps  proposed  by  Molenaar  and 
Lewis  (1979),  hereafter  M-L,  can  be  summarized  as  follows: 


Ik  -  N  <  ipk  Sf  * 


It  ~  1,  2,  m. 


where  7^  =  (  n^  x  1  )  vector  of  observed  criterion  scores  for  group 

k. 


=  (  n^  X  F  )  matrix  of  predictor  scores  in  a  set  F,  described 
below. 


=  (  n^  X  G  )  matrix  of  predictor  scores  in  a  set  6,  described 


below. 


Bp  ~  (  F  X  1  )  vector  of  unknown  regression  parameters  for  set  F 
predictors. 


B-.  =(6x1)  vector  of  unknown  regression  parameters  for  set  G 
~uX 


predictors  in  group  k, 
2 


a  =  unknown  residual  variance  for  all  m  groups, 
n^  =  number  of  individuals  in  group  k,  and 


I  =  (  n.  X  n.  )  identity  matrix. 
*  * 


In  addition,  the  unobserved  parameters  are  said  to  form  an  exchangeable 

sample  from  F  independent  uniform  distributions  for  each  variable  in  set  F. 
The  unobserved  parameters  B-.  similarly  form  an  exchangeable  sample  from  a 


N  Yq  Iq  )  distribution.  This  model  of  prior  information  is  further 

specified  by  designating  hyperparameters  and  as  exchangeable  samples 

from  D  (  -<>>,  «  )  and  inverse  chi-square  distributions  with  specified  degrees 
of  freedom,  respectively,  the  latter  in  order  to  incorporate  strengrn  of 
prior  information  into  the  model.  In  the  final  prior  specification  for  the 
2 

M-L  model.  In  o  is  assumed  to  be  uniform.  With  the  above  prior  specifica¬ 
tions  the  joint  distribution  of  parameters  and  hyperparameters  given  the 
data  is  determined — integrating  out  hype rparame ter s  yields  an  expression  for 

2 

the  joint  posterior  density  of  B_,  and  a  from  which  Molenaar  and  Lewis 

~r  —Ox 

obtain  joint  modal  estimates. 

The  M-L  model  for  regression  in  m-gronps  represents  a  general 
simplification  of  previous  Bayesian  solutions  to  the  problem  developed  by 
Lindley  (1971)  and  Lindley  and  Smith  (1972).  In  particular,  the  M-L 
specification  differs  from  the  original  formulations  in  three  important 
ways:  (1)  a  partitioning  of  predictor  variables  into  disjoint  sets,  (2)  a 

restriction  on  the  prior  between-gronp  covariance  matrix  of  the  regression 
parameters  to  diagonality,  and  (3)  a  specification  of  a  non-informative 
prior  distribution  on  a  common  residual  variance  for  all  groups.  The  im¬ 
plications  of  each  difference  for  regression  in  m-gronps  are  discussed 
below.  These  features  and  other  numerical  aspects  of  the  M-L  algorithm  lead 
to  an  accurate  and  computationally  efficient  method  for  simultaneous  estima¬ 
tion  of  multiple  regression  in  m-gronps. 

Regression  coefficients  in  the  N-L  model  are  of  two  types,  common  or 
fixed  across  groups  (the  Bp)  and  variable  across  the  k  groups  (the 

Variables  are  assigned  to  sets  F  and  G  on  the  basis  of  the  between-group 


variances  of  their  estimated  regression  coefficients.  When  prior  informa¬ 
tion  strongly  suggests  that  be  tween-group  variability  is  negligible,  a 
predictor  is  assigned  to  set  F  at  the  outset  of  the  estimation  procedure. 
Otherwise,  predictors  are  initially  assigned  to  set  G  and  are  transferred  to 
set  F  only  if  the  estimates  of  between-group  variance  fall  below  a  threshold 
value  during  the  iterative  solution.  Molenaar  and  Lewis  (1979)  describe  how 
such  estimates  are  obtained  and  used  to  partition  predictors. 

In  addition  to  circumventing  certain  problems  in  estimation  that  have 
occurred  with  previous  implementations  of  m-group  regression  models,  the 
partition  of  predictors  explicitly  recognizes  that  some  predictors  perform 
in  a  virtually  identical  fashion  across  groups.  Novick,  Jackson,  Thayer  and 
Cole  (1972)  describe  the  Lindley— Smith  model  as  one  which  seeks  a  compromise 
between  within-group  least  squares  and  pooled  least-sqnares  analyses. 
Partitioning  predictors  into  those  with  fixed  and  free  parameters  allows  for 
pooling  in  a  portion  of  the  model  when  data  and/or  prior  information  suggest 
such  pooling  to  be  appropriate.  Indeed,  when  predictor  set  G  is  empty,  the 
model  reduces  to  a  pooled  analysis,  whereas  when  set  F  is  empty  the  model  is 
equivalent  to  that  of  Lindley  and  Smith  (1972). 

A  second  feature  of  the  M-L  model  that  distinguishes  it  from  previous 
approaches  is  the  assumption  of  independent  prior  distributions  for  the 
parameters  Bp  and  Restricting  the  dispersion  matrix  for  the  to 

being  diagonal  places  rather  strong  demands  on  the  predictor  set  and  is 
likely  to  be  more  appropriate  for  some  predictor  sets  than  for  others.  As 
noted  by  Molenaar  and  Lewis,  however,  prior  knowledge  about  covariances  is 
likely  to  be  minimal  in  many  practical  situations  -  they  also  observe  that 
their  model  allows  for  such  covariances  in  the  posterior  distribution.  A 


conseqaence  of  this  aspect  of  the  model  is  that  lack  of  shrinkage  toward  a 
common  valne  across  gronps  for,  say,  will  not  influence  the  degree  of 

shrinkage  that  takes  place  for  coefficients  of  other  independent  variables. 
This  is  perhaps  reasonable  for  a  selection  battery  that  is  heterogeneous 
with  respect  to  the  abilities  required  for  test  performance,  such  as  the 
subtests  of  ASVAB. 

The  third  aspect  of  the  M-L  model  that  distinguishes  it  from  previous 
approaches  is  an  assumption  of  between-group  homoscedast  ic  ity  of  residual 
variances.  This  too  places  stronger  demands  on  data,  but  for  groups  which 
are  truly  exchangeable  such  an  assumption  may  be  no  less  unreasonable  than 
the  usual  assumption  of  homogeneity  of  variances  within-groups.  Indeed,  it 
was  observed  by  a  reviewer  that  homogeneity  of  residual  variances  between 
groups  in  the  M-L  approach  is  likely  to  be  a  serious  oversimplification  in 
practice  only  when  strong  prior  information  for  this  aspect  of  the  model  is 
available.  When  the  scaling  of  the  dependent  variable  is  arbitrary,  simple 
standardization  within  groups,  as  is  done  in  the  following  analysis,  also 
helps  to  justify  this  aspect  of  the  M-L  model. 


Method 


Data  Source 


The  M-L  model  for  m-group  regression  was  used  to  investigate  predictor- 
criterion  relationships  in  a  set  of  technical  training  data  from  the  Marine 
Corps.  The  particular  data  used  were  previously  analyzed  by  Sims  and  Hiatt 
(1981)  and  consist  of  validation  records  for  training  courses  taken  from 
general  categories  of  military  job  specialties.  Of  special  interest  is  the 
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extent  to  which  the  regressions  of  final  coarse  grade  (FCG)  in  training  on  a 
relevant  set  of  predictors  from  ASVAB  are  similar  for  a  group  of  training 
courses  considered  to  be  exchangeable.  This  is  a  special  concern  for  a 
heterogeneous  selection  battery  like  ASVAB.  A  question  that  has  plagued 
users  of  ASVAB  over  the  years  is  wnether  common  weights  for  subtests  are 
justified  for  training  programs  with  similar  content.  By  initially  con¬ 
sidering  such  programs  exchangeable,  an  alternative  assessment  of 
differences  between  regression  equations  for  subgroups  can  be  made.  The 
general  categories  of  specialties  considered  in  this  analysis  are  classified 
as  Clerical,  Electrical,  and  Mechanical.  Individual  recruits  are  assigned 
to  training  courses  on  the  basis  of  ASVAB  composite  scores  that  are  deter¬ 
mined  from  the  predictors  used  in  each  category  of  specialties. 

Data  Analysis 

The  training  courses  belonging  to  Clerical,  Electrical  and  Mechanical 
specialty  areas  are  presented  in  Table  1,  along  with  sample  sizes  for  each 
group.  Preliminary  inspections  of  bivariate  scatterplots  of  course  grades 
and  ASVAB  subtests  were  made  for  each  training  coarse  in  order  to  identify 
any  serious  departures  from  linearity  and  homoscedasticity  within  groups  and 
to  detect  outliers.  For  several  training  courses,  a  small  number  of  out¬ 
liers  were  detected  in  the  distribution  of  course  grades — such  observations 
were  deleted  in  the  ensuing  analyses  on  the  grounds  that  final  grades  for 
certain  low-performing  recruits  were  arbitrarily  determined  (see  Sims  and 
Biatt,  1981). 

For  each  category  of  training  specialties,  then,  data  analysis  con¬ 
sisted  of  initial  least-squares  regressions  of  FCG  on  the  relevant  set  of 
ASVAB  predictor  variables.  These  within-group  least  squares  results  were 


then  used  as  starting  values  in  the  M-L  simultaneons  estimation  procedure. 
All  courses  listed  in  a  given  category  in  Table  1  were  considered  exchange¬ 
able  in  the  Bayesian  analysis.  Thus,  nine  courses  were  analyzed 
simultaneously  for  the  Clerical  area,  six  for  the  Electrical  area,  and 
eleven  for  the  Mechanical  area. 


Insert  Table  1  About  Here 


The  prior  information  required  by  the  M-L  model  was  specified  in  the 
same  manner  for  the  three  types  of  specialties.  In  particular,  prior  es¬ 
timates  of  the  between-group  variance  of  the  parameters  B^^^were  obtained 

from  the  so-called  Model  II  analysis  in  a  manner  described  by  Jackson 
(1972).  In  essence,  this  method  treats  the  and  their  standard  errors 

from  the  least  squares  analysis  in  a  random-effects  ANOVA  manner  in  order  to 

derive  estimates  of  the  between-group  variance  of  P  .  for  g  =  1,  2,  ...,  G. 

8* 

These  values,  7',  were  then  treated  as  modal  estimates  from  an  inverse  chi- 
8 

square  distribution,  with  degrees  of  freedom  equal  to  1  to  indicate  minimal 
prior  information  concerning  between-group  variability  in  the  parameters. 

In  addition  to  the  separate  regression  analyses  described  above,  an 
attempt  to  understand  the  behavior  of  the  M-L  esitmates  in  future  samples 
was  made  through  a  cross-validation  study  of  the  Mechanical  specialties.  In 
this  analysis,  a  25  percent  random  sample  was  obtained  from  each  training 
course  and  used  to  estimate  parameters  by  least-squares  and  M-L  methods. 
The  estimates  obtained  from  these  samples  were  used  in  predicting  course 
grades  of  recruits  in  the  remaining  75  percent.  It  should  be  clear  that 
this  procedure  does  not  mirror  exactly  an  ideal  cross-validation  study. 
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Nevertheless,  it  does  provide  a  beginning  to  understanding  how  the  M-L 
estimation  procedure  might  be  expected  to  perform  in  practice,  especially 
for  training  programs  with  sample  sizes  that  would  otherwise  prohibit 
separate  least-squares  solutions. 


Results 

The  principal  results  presented  are  the  estimates  of  regression 
parameters  based  on  least-squares  and  M-L  nt-group  analyses.  The  dependent 
variable,  FCG,  has  been  standardized  within-gronps  to  remove  apparent  dif¬ 
ferences  between  training  courses  in  grading  standards  from  the  criterion 
distributions.  The  independent  variables,  ASVAB  snbtests,  are  typically 
reported  on  scales  ranging  from  20  to  80  and  exceptions  to  this  are  noted  in 
the  description  of  results. 

Clerical  Specialties 

ASVAB  subtests  used  in  the  selection  composite  for  clerical  specialties 
include  ability  tests  of  Arithmetic  Reasoning  (AR) ,  Word  Knowledge  (WK) , 
At t e n t i on- to-De t a i 1  (AD),  and  an  attitudinal  measure  called  the 
Attentiveness  Scale  (CA) .  Unlike  scores  for  the  ability  measures,  observed 
scores  on  CA  can  range  from  0  to  20.  The  results  of  within-group  least- 
squares,  pooled  least-squares,  and  M-L  analyses  are  summarized  in  Table  2. 
Estimates  of  coefficients  for  the  four  independent  variables  appear  under 
the  appropriate  column  heading.  Rather  than  reporting  the  estimated 
intercept  at  0,  which  is  out  of  range  on  the  joint  predictor  distribution. 


the  intercept  at  the  pooled  centroid  of  the  predictors  is  reported  under  the 


heading  Int(C).  This  valne  allows  a  more  suitable  comparison  of  any  inter¬ 
cept  differences  that  may  exist  among  the  groups.  The  residual  standard  | 

deviations  for  the  least-squares  analysis  appear  in  the  column  marked  Res 
SD. 

The  within-group  least-squares  results  in  Panel  (a)  show  clear  dif¬ 
ferences  among  the  groups,  both  with  respect  to  intercept  and  slopes  of  the 
regression  surfaces.  Notable  features  of  these  results  include  the  pattern 
of  positive  and  negative  intercepts  across  groups  and  the  weights  of  rela¬ 
tively  small  magnitude  for  AD  (recall  AD  is  scaled  in  the  same  way  as  are  AR 
and  WK)  .  In  addition,  coefficients  estimated  for  the  attitndinal  measure, 

CA.  display  marked  variation  among  the  groups.  However,  when  one  considers  j 

that  typical  standard  deviations  on  .ais  measure  are  2.5  to  3  points,  the 
contributions  made  by  it  to  prediction  are  quite  small.  Indeed,  the  usual 
significance  tests  failed  to  reject  the  null  hypothesis  that  the  coeffi¬ 
cients  for  both  AD  and  CA  were  zero  at  the  .05  level  for  all  Clerical 
specialties.  Nevertheless,  these  variables  were  included  in  the  m-group 
analyses  in  part  to  monitor  the  extent  to  which  between-group  differences  on 
these  variables  were  due  to  sampling  fluctuations.  Although  not  included  in 
the  table,  multiple  correlations  in  the  least-squares  analysis  ranged  from 
.40  to  .79  within  groups  (.59  in  the  pooled  sample). 


Insert  Table  2  About  Here 
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specified.  Estimates  of  coefficients  for  AR  and  VZ  do  not  differ  to  any 
important  degree  across  the  nine  specialties  and  the  apparently  large  dif¬ 
ferences  observed  for  coefficients  of  CA  in  the  least  squares  analysis  are 
seen  as  a  consequence  of  sampling  variation  through  the  eyes  of  the  Bayesian 
approach.  Though  not  reported  here,  results  for  the  M-L  model  with  predic¬ 
tors  AD  and  CA  removed  were  very  similar  to  those  in  Panel  (b),  with  only  a 
small  increase  in  the  residual  SD  estimate  caused  by  the  reduced  predictor 
set . 

Where  clerical  specialties  do  differ,  even  in  the  M-L  solution  ,  is  in 
their  intercepts  at  the  pooled  centroid.  Application  of  the  M-L  model 
didn't  greatly  influence  the  intercept  differences  noted  in  the  least 
squares  solutions.  Aside  from  this  factor,  the  ASVAB  subtests  used  for 
clerical  specialties  perform  quite  consistently  in  predicting  course  grades. 
Justification  for  differential  weighting  of  predictors  among  training 
courses  would  apparently  have  to  come  from  an  assumption  that  some  courses 
are  not  exchangeable  in  the  way  specified  by  the  M-L  model. 

Electrical  Specialties 

ASVAB  subtests  used  in  selection  for  courses  classified  as  electrical 
specialties  were  AR,  General  Science  (GS),  Mathematics  Knowledge  (MK),  and 
Electrical  Information  (El).  Results  of  regression  analyses  from  the 
various  approaches  are  given  in  Table  3,  the  contents  of  which  parallel 
those  of  the  previous  table. 

The  least-squares  estimates  for  Electrical  specialties  show  greater 
variation  among  groups  than  was  seen  in  the  case  of  Clerical  specialties. 
Multiple  correlations  for  this  group  of  specialties  ranged  from  .15  to  .58 
(.37  in  the  pooled  sample).  Differences  between  groups  are  particularly 


noticeable  for  coefficients  of  AR,  which  are  relatively  large  for  Avionics 
Repair.  Basic  Electrician  and  Basic  Electronics,  and  near  zero  for  the 
remaining  courses.  Moreover,  the  least~sqnares  coefficient  for  MK  in  the 
Basic  Electronics  group  is  much  larger  (.047)  than  it  is  in  any  other  group. 
In  contrast  to  results  from  the  Clerical  specialties,  no  single  predictor 
variable  in  the  least-squares  analysis  appears  less  important  than  the 
others  in  predicting  performance,  at  least  based  on  the  magnitudes  of  the 
regression  weights.  Again,  because  the  immediate  purpose  here  is  not  vari¬ 
able  selection,  all  subtests  are  retained  for  the  M-L  analysis. 


Insert  Table  3  About  Here 


The  M-L  results  in  Panel  (b)  again  show  regression  toward  a  common 
value  for  many  of  the  coefficients  in  the  model  used  with  Electrical 
specialties.  One  predictor,  AR,  shows  much  greater  homogeneity  across 
groups — the  Bayesian  estimates  of  weights  for  this  variable  are  also  quite 
different  in  some  cases  from  the  pooled  least-squares  weights  given  in  Panel 
(a).  Note  also  that  the  weight  for  the  Electrical  Information  test  (El)  was 
judged  constant  across  groups  using  the  Model  II  prior  estimates  of  between- 
gronp  variances.  A  contrast  to  this  degree  of  homogeneity  is  observed  with 
respect  to  predictors  GS  and  MK.  Estimated  weights  of  the  former  range  from 
.014  to  .021,  while  those  of  the  latter  are  around  .026  for  all  but  the 
Basic  Electronics  course,  whose  estimated  weight  under  the  M-L  model  was 
.046.  As  seen  in  the  results  for  Clerical  specialties,  intercepts  for  the 
six  Electrical  training  courses  are  quite  distinct  when  evaluated  at  the 
centroid  of  the  pooled  distributions.  With  small  mean  differences  on  the 
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predictors  known  to  exist  for  these  groups,  this  again  is  an  unsurprising 
result . 

Although  estimates  of  slopes  for  the  six  Electrical  specialties  were 
quite  similar  for  two  predictors,  even  the  H-L  results  fail  to  justify  a 
single  prediction  equation  for  all  specialties  in  this  category.  Predicting 
success  for  the  Basic  Electronics  group  using  this  set  of  predictors  clearly 
requires  heavier  weight  to  be  placed  on  MK.  Whether  such  a  result  is  taken 
to  mean  that  Basic  Electronics  is  not  exchangeable  with  the  other  Electrical 
specialties  is  perhaps  open  to  question.  The  H-L  results  indicate  that  even 
when  exchangeability  is  assumed  a  priori,  the  data  warrant  that  a  prediction 
model  for  this  course  be  considered  separately  from  those  of  other 
Electrical  specialties. 

Mechanical  Specialties 

The  ASVAB  subtests  that  belong  to  the  selection  composite  for  mechani¬ 
cal  specialties  are  again  AR  and  GS.  used  previously,  a  test  of  Mechanical 
Comprehension  (MC)  and  a  test  of  Automotive  Information  (AI)  .  Results  of 
the  regression  analyses  using  these  subtests  as  predictors  are  given  in 
Table  4. 

Variation  from  group  to  group  in  the  magnitudes  of  least-squares 
regression  weights  is  again  the  rule  rather  than  the  exception  for  the 
Mechanical  specialties.  With  respect  to  GS,  weights  are  near  zero  for  the 
Aviation  Crash  Crew  and  Small  Arms  Repair  courses,  yet  of  substantial  mag¬ 
nitude,  relatively  speaking,  for  ASM  (Structures)  and  Tracked  Vehicle  Repair 
(.034  and  .043,  respectively).  The  other  predictor  in  this  set  that  dis¬ 
plays  marked  variation  in  weights  across  groups  is  AI,  which  has  a  near  zero 
weight  for  ASM  (Safety)  and  a  clearly  non-zero  weight  for  the  two  automotive 
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mechanics  training  courses.  The  magnitudes  of  weights  assigned  to  AR  and  MC 
are  much  more  homogeneous  in  the  least-squares  analyses  —  indeed,  the 
estimate  given  for  MC  the  pooled  sample  is  quite  representative  of  nearly 
all  within-group  estimates.  The  pattern  of  positive  and  negative  intercepts 
at  the  pooled  centroid  is  again  seen  in  the  results  for  mechanical 
specialties,  as  is  some  variability  in  the  size  of  the  standard  errors  of 
estimate.  Multiple  correlations  for  these  groups  ranged  from  .34  to  .67, 
with  a  value  of  .50  obtained  in  the  pooled  sample. 


Insert  Table  4  About  Here 


Shrinkage  of  parameter  estimates  toward  common  values  in  the  M-L  ap¬ 
proach  is  again  observed  in  the  results  in  Panel  (b)  of  Table  4.  Two 
variables  (AR  and  MC)  were  assigned  to  predictor  set  F  on  the  basis  of  prior 
specifications  determined  from  the  Model  II  analysis.  However,  the  M-L 
estimates  of  parameters  for  predictors  GS  and  AI  have  only  moderately  ap¬ 
proached  a  value  that  is  common  across  groups.  Although  the  coefficient  for 
GS  in  the  Tracked  Vehicle  Repair  course  has  become  closer  in  value  to  those 
of  other  courses,  weights  for  GS  are  still  comparatively  small  in  the  Crash 
Crew  and  Small  Arms  Repair  courses.  Moreover,  GS  appears  to  play  a  more 
prominent  role  in  predicting  course  grades  in  the  Advanced  Auto  course  than 
it  does  in  the  Basic  Auto  course.  These  differences  were  still  manifest 
when  prior  specifications  were  altered  to  indicate  that  more  weight  should 
be  placed  on  the  Model  II  analysis.  Given  the  strong  assumptions  of  the  M- 
L  model,  differences  like  these  would  be  difficult  to  ignore  in  future 
specifications  of  prediction  equations  for  these  courses.  Other  between- 
group  differences  that  remain  even  after  application  of  the  M-L  approach 
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involve  estimates  of  intercepts  and  of  weights  for  AI,  which  remain  larger 
for  the  two  automotive  training  courses. 

Cross  Validation 

An  additional  concern  when  results  of  a  series  of  analyses  like  those 
in  this  report  are  to  be  used  for  future  versions  of  an  aptitude  battery  is 
the  expected  stability  of  regression  coefficients  on  cross-validation.  The 
issues  relevant  to  this  question  have  received  much  attention  in  the  litera¬ 
ture  over  the  years  and  no  review  will  be  given  here.  Bayesian  methods  for 
simultaneous  estimation  of  regression  coefficients  have  been  shown  to  cross- 
validate  better  than  within-group  least-squares  (cf.  Novick.  Jackson,  Thayer 
and  Cole,  1972),  particularly  with  small  sample  sizes.  This  result  was 
confirmed  for  the  Molenaar-Lewis  approach  with  the  limited  cross-validation 
study  performed  on  data  from  the  (fecfaanical  specialties.  Table  5  contains 
mean-squared  errors  (MSE)  and  correlations  (CORK)  between  observed  and 
predicted  criterion  scores  from  the  cross-validation  analysis.  The  results 
in  Table  S  are  generally  consistent  with  past  comparisons  of  Bayesian  m- 
group  techniques  and  conventional  methods  —  a  small  yet  consistent  trend 
toward  smaller  errors  of  prediction  on  cross-validation  using  a  Bayesian  m- 
group  model.  Although  the  differences  between  least-squares  and  M-L  errors 
given  in  Table  5  are  quite  small  —  absolute  differences  between  MSB's 
ranging  from  .001  to  .043  —  this  is  perhaps  to  be  expected  when  the  cross- 
validation  sample  represents  data  from  the  same  year  as  the  calibration 
sample.  If  one  goal  of  the  Bayesian  method  is  to  smooth  out  minor  temporal 
fluctuations  in  the  parameter  estimates  that  might  otherwise  be  interpreted 
as  differences  between  groups,  then  one  would  expect  greater  accuracy  on 
cross-validation  for  the  M-L  estimates  and  data  from  a  subsequent  year. 


Insert  Table  5  About  Here 


Discussion 


Application  of  the  M-L  model  for  m-group  regression  to  the  prediction 
of  success  in  technical  training  generally  snpports  the  use  of  common 
weights  when  ASVAfi  subtests  are  used  to  construct  selection  composites.  If 
one  were  to  place  heavy  reliance  on  the  results  of  the  within-group  least- 
squares  analyses,  a  different  conclusion  would  certainly  follow  from  a 
simple  examination  of  estimated  coefficients,  even  with  sample  sizes  as 
large  as  those  available  in  this  data  set.  To  the  extent  that  the  assump¬ 
tion  of  exchangeability  is  satisfied  by  the  groups  analyzed  simultaneously, 
the  M-L  results  provide  a  useful  alternative  assessment  of  the  differences 
between  specific  training  programs  with  similar  content.  These  differences 
were  found  to  be  negligible  for  the  group  of  Clerical  training  programs 
considered,  but  of  sufficient  magnitude  for  certain  Electrical  and 
Mechanical  specialties  to  warrant  more  careful  consideration  when  selection 
composites  for  future  versions  of  ASVAB  are  developed. 

A  consideration  of  utmost  importance  in  evaluating  the  appropriateness 
of  the  M-L  model  for  developing  prediction  equations  for  technical  training 
specialties  in  the  military  is  the  question  of  exchangeability.  The  ap¬ 
proach  to  the  question  adopted  in  this  paper  has  been  to  assume 
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exchangeability  among  training  courses  on  the  basis  of  coarse  content  and  to 
allow  resnlts  to  point  to  groups  which  might  well  be  distinct.  Deletion  of 
the  few  specialties  in  the  Electrical  and  Mechanical  areas  that  seem  atypi¬ 
cal  of  the  area  at  large  would  no  doubt  produce  even  greater  homogeneity  of 
regression  coefficients  for  predictors  than  has  been  reported  here. 
However,  more  experience  in  applying  the  M-L  method,  or  similar  methods,  to 
data  from  other  recruiting  years  is  likely  to  provide  a  better  check  on  the 
extent  to  which  exchangeability  is  justified  for  the  groups  studied  in  this 
analysis.  In  general,  it  seems  that  this  type  of  assumption  is  properly 
evaluated  over  time  rather  than  at  a  specific  point  in  time. 

The  choice  of  the  Mo  1 enaar-Lewi s  model  for  m-group  regression  also 
receives  some  support  from  the  cross-validation  results.  As  observed  in  the 
description  of  the  model,  M-L  places  greater  restrictions  on  the  specifica¬ 
tion  of  prior  information,  in  part  to  increase  computational  efficiency  and 
to  avoid  certain  estimation  problems  (Molenaar  and  Lewis,  1979,  pp.  6ff.). 
These  restrictions  do  not  appear  to  have  compromised  the  effectiveness  of 
the  model  for  technical  training  specialties  in  the  Marine  Corps.  Whether 
or  not  a  model  with  more  detailed  prior  specifications  would  yield  results 
that  differ  perceptibly  from  those  of  the  M-L  approach  is  an  open  question  - 
-  the  extent  of  improvement  would  certainly  be  related  to  the  strength  of 
that  additional  prior  information.  It  is  far  from  obvious  that  strong  prior 
information  concerning,  for  example,  between-group  covariances  of  regression 
parameters  or  between-group  variances  of  residual  standard  deviations  is 
available  for  military  training  specialties  at  the  present  time.  Further 
study  of  such  specialties  using  m-group  techniques  should  certainly  consider 
applying  more  detailed  prior  specifications  and  methods  of  estimating  the 
required  hyperparameters.  Some  informal  comparisons  made  with  data  of  the 
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this  study  indicate  N-L  yields  results  similar  to  those  from  a 
f  Rubin's  (1980)  empirical-Bayes  approach  when  the  H-L  analysis 
after  standardizing  the  criterion  variable  within  groups. 

Conclusion 


Application  of  the  Molenaar-Lewis  model  for  regression  in  m-groups  to 
the  problem  of  predicting  training  success  in  various  Marine  Corps  job 
specialties  indicates  some  justification  for  limited  use  of  common  weights 
for  predictor  variables  in  training  courses  considered  exchangeable  on  a. 
priori  grounds.  All  groups  in  the  Clerical  area  were  characterized  by 
slopes  of  similar  magnitude,  although  intercept  differences  were  common. 
For  both  Electrical  and  Mechanical  specialty  areas,  training  courses  were 
identified  that  had  estimated  slopes  differing  markedly  with  respect  to  at 
least  one  of  the  predictor  variables  included.  Continued  monitoring  of  such 
courses  is  important  in  judging  the  appropriateness  of  a  common  prediction 
equation  for  all  training  programs  in  these  two  areas. 

The  relevance  of  the  methodology  of  si-group  regression  to  predicting 
success  in  a  variety  of  military  training  programs  is  an  important  outcome 
of  this  analysis.  The  extreme  views  of  complete  generalization  of  the 
criterion-related  validity  of  ASVAB  subtests  across  all  courses  and  of 
entirely  course-specific  characterizations  of  snbtest  validity  are  equally 
unattractive.  The  model  for  m-group  regression  used  in  this  study  allows  an 
assessment  of  exactly  where  between  these  two  extreme  positions  an  accurate 


characterization  of  criterion-related  validity  lies. 
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Sample  Sizes  for  Marine  Corps 
Specialty  Areas 


Specialty  Area  Sample  Size 

Clerical 


Basic  Supply  Stock  1238 
Personal  Financial  Records  375 
Administrative  1336 
Personnel  176 
Unit  Diary  149 
Communications  Center  711 
Aviation  Operations  247 
Aviation  Maintenance  Administration  215 
Aviation  Supply  496 

Electrical 

Basic  Electrician  223 
Electrical  Equipment  Repair  215 
Basic  Electronics  1093 
Radio  Fundamentals  1^5 
Field  Radio  Operator  1244 
Avionics  Repair  297 

Mechanical 

Basic  Auto  Mechanic  1276 
Advanced  Auto  P<(echanic  618 
Combat  Engineer  934 
Engineering  Equipment  Mechanic  691 
Tracked  Vehicle  Repair  233 
Basic  Helicopter  801 
ASM*  (Safety)  124 
ASM*  (Hydraulics)  563 
ASM*  (Structures  611 
Aviation  Crash  Crew  295 
Small  Arms  Repair  323 


*ASM  =  Aviation  Structural  Mechanics 


Table  2 
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Table  3 

Least-squares  and  M-L  m-group  Estimates  of 
Regressions  for  Electrical  Specialties 

Panel  (a)  -  Least-squares 


Training  Course 

Int(C) 

AR 

GS 

MR 

El 

Res  SD 

Basic  Electrician 

.228 

.019 

.014 

.023 

.026 

.930 

Elec.  Equip.  Repair 

-.113 

.007 

-.004 

.031 

.018 

.968 

Basic  Electronics 

-.354 

.019 

.022 

.047 

.021 

.813 

Radio  Fundamentals 

-.385 

.009 

.030 

.026 

.012 

.965 

Field  Radio  Operator 

.299 

.009 

.017 

.028 

.017 

.914 

Avionics  Repair 

-.436 

.035 

.025 

.012 

.034 

.922 

Pooled 

.000 

.003 

.013 

.027 

.014 

.931 

Panel 

(b)  - 

Molenaar- 

-Lewis 

Training  Coarse 

INT(C) 

AB 

GS 

HR 

El* 

Basic  Electrician 

.212 

.015 

.018 

.026 

.020 

Elec.  Equip.  Repair 

-.144 

.014 

.014 

.026 

.020 

Basic  Electronics 

-.329 

.015 

.021 

.046 

.020 

Radio  Fundamentals 

-.387 

.014 

.019 

.027 

.020 

Field  Radio  Operator 

.314 

.014 

.016 

.026 

.020 

Avionics  Repair 

-.318 

.015 

.021 

,025 

.020 

Modal  Estimates  of  Res. 

,  SD  = 

.888 

*EI  was  judged  to  belong  to  set  F  using  the  Model  II  prior  estimate  of 
between-group  variance. 
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Princeton,  MJ  08451 
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Dr.  liarii  D.  Rs:f;25e 


G.  Box  liB 


;  D^.  Sasuel  T.  ^3,' 

Lovoia  Univsraiiv  at  Chicago 
320  North  Wich’can  Avenue 
Chicago.  IL  3Coll 

1  silliij  J.  IcLaurir 
iifilO  Howis  Court 
Cai;  Springs.  ’’D  COCCI 

1  C’".  Ba’'Dar2  Nears 
-i-.''ar:  ■■esourcss  Ssasarcr)  Crgannaticn 
COO'  No'-tr  iiasiiirgton 
Alexandria.  0*  CCCIA 

i  Dr.  Robert  Nislevy 
’!!  niincia  Street 
Sensva,  IL  SOICA 

1  D,'.  Nelvin  S.  Novic»: 

C5s  Liidouist  Center  #or  Neasoraent 
L'ni'versitv  of  Ic«a 
iOrtS  City,  lA  '2242 

1  Dr.  Jaaes  Dlscn 

:S7j  South  'State  Street 
Drsi.  'JT  34057 

;  Dr.  Jesse  Drlaosky 
Iristit'jte  for  Defense  Analyses 
!:01  N,  Beauregard  St. 

Alexandria,  Va'cCCI! 

:  ,ia,'s  N.  Fatience 
Anerican  Council  on  Education 
BED  testing  Seryice,  Suite  20 
'One  Dupont  Cirle.  NN 
Washington,  DC  20036 

I  Dr.  James  A.  ^aulson 
Portland  State  University 
'.D.  Box  751 
»ortland,  OR  ?’20:’ 

1  Dr.  Sam  Pearson 
DACI.  Inc.  FED. 

Department  5R21 

1S15  North  Fort  Nvers  Drive 

■Arlington,  VA  22209 


Dr.  Thcsas  Reynolds 
amversitv  ot  Tsxas-Dallas 
.Marketing  Department 
P.  0.  Box  688 
Richardson,  TX  750S0 

Nr,  .<en  Riecx 
FESSIN/NA 

6118  Franconia  Road 


Dr.  Andrei*  N.  Rose 
Aiterican  I.nstitutes  for  Resea 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

Dr.  J,  Ryan 

Department  of  Education 
University  of  South  Carolina 
Columbia.  SC  29208 

PRCF.  FUNIKC  SAKEJINA 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  TENNESSEE 
KNOXVILLE.  'N  37916 

Frank  L.  Schmidt 
Department  of  Psychology 
Bldg.  SS 

Beorgg  Washington  university 
Washington,  DC  20C52 

Lowell  Sc.hoer 

F'sycholcgica!  i  Ouantitative 
Foundations 
College  of  Education 
University  of  Iowa 
Iowa  City,  IR  52242 

Dr.  Kazuo  Shigeiasu 
7-9-24  Kugenufla-kaigan 
F'jjusawa  251 
JAPAN 

Dr.  Edwin  Shirkev 
Deoartsent  of  *^5vchoIagv 
Unrversity  of  Central  Florida 
Orlando,  FL  32316 


1  Dr.  «’Il;2!n  riis 
Ce.'tsr  tor  Naval  Analysis 
ZOO  North  Beaursqaro  Street 
Aleyaodria,  V,A  2231! 


Dr.  Hoiiard  aaioer 
Division  ct  ’Psychological  Studies 
Educational  Tasting  Service 
^’rinceton,  NJ  DS540 


1  Dr.  H,  Nallace  Si.nauo 
Frograi  Director 

ilanpoHer  Research  and  Advisory  Services 
Ssithsonian  Institution 
30!  North  Pitt  Street 
Alexandria,  vA  22314 

1  Dr.  Peter  StolotT 
Center  Por  Naval  Analysis 
200  North  Beauregard  Street 
ilexandna.  VA  2231! 

1  2r.  aillias  Stout 
tniversity  of  Illinois 
Departsent  of  Nathesatics 
urbana,  IL  6130. 

1  Dr.  Hariharan  Swainathan 
laboratory  of  Psychoaetric  and 
Evaluation  Research 
School  of  Education 
University  of  hassacnusects 
Aiiherst,  1A  01003 

1  Dr.  Kikusi  Tatsuoka 
Dc.oDuter  Based  Education  Research  Lab 
2S2  Engineering  Research  Laboratory 
Urtana,"  IL  ilSOl 

1  Dr.  Naurice  Tatsuoka 
220  Education  Bldg 
1310  S.  Sixth  St. 

Chainijaign,  IL  31320 

1  Dr.  David  ■''hissen 
Department  of  Psychology 
.University  of  Kansas 
Laurence,  KS  66044 


1  Dr.  Brian  Naters 
.HumRRC 

300  North  Washington 
Alexandria,  VA  22314 

1  3r.  David  J.  Weiss 
N660  Elliott  Hall 
University  of  Minnesota 
75  E.  River  Road 
Minneapolis,  MN  55455 

!  Dr.  Donald  fl.  Neitcman 
Mitre  Corporation 
1320  Dcilev  Madison  Elvd 
McLean,  VA  22102 

1  DR,  SERSHON  WELT.MAN 
PERCEPTRQNICo  INC. 

6271  VARIEL  AVE. 

WOODLAND  HluLS,  DA  91367 

1  Dr.  Rand  R.  Wilcox 
University  of  Southern  California 
Department  of  Psychology 
Los  Angeles,  CA  90007 

1  Wolfgang  Wildgrube 
Streitkraefteaot 
Box  20  50  03 
0-5300  Bonn  2 
WEST  SERHANY 

1  Dr.  Bruce  Wi Ilians 
Department  of  Educational  Psvcholog 
University  of  Illinois 
Urbana,  IL  61801 


1  Dr.  Robert  Tsutakawa 
Department  of  Statistics 
University  of  .Missouri 
Columbia,  MO  65201 

1  Dr,  Da /id  Vale 
Assessment  Systems  Corporation 
2233  University  Avenue 
Suite  310 

St.  Paul.  MN  55114 
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Additional  Names 

1  Dr.  Frank  Erwin 
President 

Richardson,  Bellows,  Henry  &  Co. 
Suite  612 

1140  Connecticut  Avenue,  NW 
Washington,  DC  20036 

1  Dr.  Lorraine  Eyde 

Office  of  Personnel  Research  & 
Development  Center 
1900  E  Street,  NW 
Washington,  DC  20415 

1  Dr.  Milt  Hakel 
Ord  Inc. 

2455  N.  Star  Road 
Suite  303 

Columbus,  OH  43221 

1  Dr .  Samuel  Messick 

Vice  President  for  Research 
Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr,  Nancy  Petersen 

Educational  Testing  Service 
Princeton,  NJ  08541 

1  Dr.  Mary  Tenopyr 
AT  &  T 

550  Madison  Avenue 
Room  1141 

New  York,  Ni  10022 

1  Dr.  Alexandra  Wigdor 
National  Research  Council 
2101  Constitution  Avenue,  NliJ 
Washington,  DC  20418 
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